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ABSTRACT 


A  novel  associative  memory  is  dj4cussed  whiV:h  overcomes 
the  early  saturation  problem  of  Hopfield  memories,  without 
resorting  to  dilute  state  vectors  or  nonlocal  learning  rules. 

/ 

The  mem  ~y  uses  a  Bidirectional  Linear  Transformer  (BLT) 
which  transfor  .  the  bipolar  inp.dt  vector  x  into  a  vector  u, 
which  is  a  lint.  :  combination  of  Hadamard  vectors.  The  matrix 
of  the  BLT  is  of  Hebbian  form,  equal  to  the  sum  of  outer 

products  of  stored  vectors  q  and  Hadamard  vectors  h  .  The 

oc  ______  _ — - - - - - 


Hadamard  vectors  arp  considered  to  serve  as  labels  for  the 
scored  vectors.  The  BLT  is  followed  by  a  Dominant  Label 
Selector  (DLS),  which  finds  the  dominant  Hadamard  component 
in  the  linear  combination  u,  and  returns  the  associated 
Hadamard  vector  to  the  BLT,  to  be  processed  in  the  BLT  back- 
stroke.  Thjs  backstroke  produces  the  stored  vector  closest  to 
the  input  x.  The  maximum  number  of  stored  vectors  that  can  be 
perfectly  retrieved  by  associative  recall  is  equal  to  the 
dimension  N  of  the  BLT  and  DLS. 


The  present  report  deals  with  the  DLS,  which  may  be  seen 
as  an  associative  memory  which  stores  N  orthogonal  bipolar 
vectors,  the  Hadamard  vectors.  A  DLS  architecture  has  been 
found  which  gives  perfect  associative  recall  of  these  stored 
vectors.  The  method  involves  a  quadratic  activation  which,  on 
account  of  a  group  property  of  Hadamard  vectors,  requires  no 
more  physical  connections  than  a  fully  connected  Hopfield 
memory  of  the  same  dimension. 

/  /  VI 

The  dynamics  of  this  "quadratic  Hadamard  memory"  is 
investigated  in  the  asynchronous  discrete  model.  Stability  is 
assured,  and  it  is  shown  in  a  long  but  simple  proof  that  the 
stable  states  of  the  memory  are  the  Hadamard  states  and  no 
others . 


Computer  simulations  performed  for  dimension  N=16 
are  in  agreement  with  the  theory  developed. 


INTRODUCTION 


Associative  memories  are  to  perform  associative  recall 
of  stored  vectors.  The  simplest  such  memory  that  has  fault 
tolerance  is  the  Hopfield  memory  [1],  in  which  the  connection 


matrix  is  the  sum  of  the  outer  products  of  the  stored  vectors, 

with  the  diagonal  removed.  The  problem  with  such  memories  is 

early  saturation.  This  problem  can  be  circumvented  by  using 

vectors  with  dilute  information  [2],  or  by  employing  a  more 

sophisticated  connection  matrix  [3,4],  The  former  of  these 

approaches  makes  inefficient  use  of  dimension,  and  the  latter 

requires  non-local  learning  rules,  which  complicate  hardware  Codes 

implementations.  Bidirectional  Associative  Memories  [5-7]  by  . - 
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themselves  do  not  solve  the  early  saturation  problem  either 

[8]  . 


The  present  DARPA  SBIR  project  is  an  attempt  to  find  an 
approach  to  associative  memories  that  overcomes  the  early 
saturation  problem  in  such  a  manner  that  efficient  use  is 
made  of  dimension,  local  learning  can  be  used,  and  hardware 
implementations  are  simple.  In  Phase  I  of  the  project  an 
approach  was  outlined  that  showed  promise  for  reaching  these 
objectives.  It  uses  a  modification  of  the  Bidirectional 
Associative  Memory  (BAM)  in  which  the  rear  threshold  opera¬ 
tion  is  replaced  by  something  we  have  called  a  Dominant  Label 
Selector  (DLS).  The  idea  is  to  choose  backstates  of  the  BAM 
as  orthonormal  vectors,  which  we  consider  to  be  labels  for 
the  stored  frontstates.  For  any  input  state,  the  backstate  of 
the  BAM  is  then  a  linear  combination  of  these  labels,  and  the 
task  of  the  DLS  is  to  find  the  dominant  label  in  this  linear 
combination,  and  return  it  to  the  BAM,  to  be  used  in  its 
backstroke.  It  is  easy  to  see  that  the  device  performs  per¬ 
fect  associative  recall  if  the  DLS  works  as  required.  To  find 
such  a  DLS  is  the  main  challenge  of  the  project. 

A  solution  to  this  problem  has  been  found  in  the 
present  Phase  II.  The  DLS  conceived  is  an  associative  memory 
with  an  activation  that  is  quadratic,  rather  than  the 
customary  linear  (really,  affine)  function  of  the  neuron 
signals.  This  approach  falls  under  what  is  known  as  "higher- 
order  neurons",  which  has  been  discussed  in  broad  terms  in 
the  literature  [9-12].  There  is  the  immediate  practical 

4 

question  about  the  large  number  of  connections  (N  )  that 
appears  to  be  required  in  such  a  case,  but  fortunately,  the 
number  of  connections  is  found  to  have  an  upper  bound  of  only 

A 

N,  as  for  a  Hopfield  memory.  The  other  practical  question 
pertains  to  the  hardware  implementation  of  the  signal  product 
operation  required  in  computing  the  quadratic  activation. 

This  question  needs  further  work,  but  it  should  be  mentioned 
already  that  we  see  a  simple  implementation  by  means  of  a 
field  effect  transistor  operating  in  the  unsaturated  region. 

The  DLS  acts  on  linear  combinations  of  orthonormal 
labels,  and  returns  the  dominant  label.  Since  the  associative 
memory  considered  is  bipolar,  the  orthonormal  labels  are  bi¬ 
polar.  The  matrix  of  a  complete  orthonormal  set  of  bipolar 
vectors  is  known  as  a  Hadamard  matrix  [13],  The  DLS  con¬ 
sidered  therefore  may  be  called  a  "quadratic  Hadamard  memory". 
This  report  is  about  such  memories.  The  questions  discussed 
are  stability  and  stable  states,  and  the  number  of  physical 
connections.  These  questions  are  investigated  in  the  asyn¬ 
chronous  discrete  model. 

In  regard  to  conventions  and  notations,  following  Kosko 
[14],  we  call  the  input  and  output  of  the  threshold  function 
respectively  the  "activation"  and  the  "signal"  of  the  neuron. 
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If  a  neuron  does  not  perform  thresholding,  then  the  activa¬ 
tion  is  defined  as  the  result  of  the  summing  operation  done 
by  the  neuron. 

The  vectors  x  and  y  are  bipolar,  i.e.,  all  their  compo¬ 
nents  are  either  1  or  -1. 

In  order  to  keep  formulas  simple,  we  have  used  the  sum¬ 
mation  convention  of  tensor  calculus  where  convenient.  In 
order  to  distinguish  from  unsummed  expressions,  the  summation 
convention  has  been  used  in  its  strict  form  [15] :  in  a 
product,  summation  over  a  repeated  index  is  implied  only  if 
the  index  appears  once  as  a  subscript,  and  once  as  a  super- 

a 

script.  For  instance,  u  v  is  summed,  but  u  v  is  not. 

Indices  are  used  as  follows,  i,  j,  and  k  denote  compo¬ 
nents  in  input  space,  a,  b,  c  and  p  denote  components  in  the 
space  between  the  front  end  and  the  DLS ,  and  also  components 
of  DLS  state  vectors.  ryC ,  p  ,  and  Jr  are  used  to  name  stored 
vectors  and  Hadamard  vectors.  All  indices  range  from  1  to  N. 

Indices  are  raised  and  lowered  with  the  Kronecker  delta 

CL 

as  metric  tensor,  hence  v  and  v  have  the  same  numerical 
value.  *- 

As  a  further  simplification  of  appearance,  1  is  often 
written  as  +  and  -1  as  -,  when  no  confusion  with  composition 
symbols  Coin  arise. 

In  the  theorems,  V  means  "for  all",  3  means  "such  that", 
means  "implies",  and  means  "is  implied  by". 

S3 means  "there  exists". 


SELECTIVE  REFLEXIVE  MEMORY  ( SRM ) 

The  associative  memory  under  discussion  consists  of 
a  front  end  followed  by  a  Dominant  Label  Selector  (DLS).  The 
front  end  is  much  like  a  Bidirectional  Associative  Memory 
(BAM)  [5].  However,  there  is  no  rear  thresholding,  and 
therefore  the  device  is  given  a  separate  name, 

"Bidirectional  Linear  Transformer"  (BLT).  The  BLT  has  thres¬ 
holding  of  frontstates,  as  in  the  BAM.  The  whole  machine,  BLT 
plus  DLS,  is  called  "Selective  Reflexive  Memory"  (SRM),  in 
which  "selective"  indicates  the  selection  of  the  dominant 
label  by  the  DLS,  and  "reflexive"  alludes  to  the  signals 
passing  back  and  forth  through  the  BLT.  The  SRM  is  shown  in 
Fig.  1. 


In  the  discrete  model,  the  BLT  performs  a  linear  trans¬ 
formation  in  both  forward  and  backstrokes,  the  forward  stroke 
being 


V  \ 


m 
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where  the  bipolar  vector  x,  i=  1  to  N,  is  the  input  vector, 

t- 

and  u^,  b=l  to  N,  is  the  ouput  of  the  3LT.  The  backstroke 
through  the  BLT  gives 


w 


v  H 


(2) 


m  which  y  is  the  bipolar  vector  returned  by  the  DLS  to  the 


back  of  the  BLT,  and  w^.'s  the  result  of  the  backstroke, 
appearing  in  front  of  the  BLT.  Subsequently,  w.  is  thresholded, 

and  the  result  is  a' bipolar  vector  x .  There  is  the  option  of 


using  x  [  to  upgrade  the 

c 


input 


x .  , 

L 


as  depicted  in  Fig.  2a,  or 


to  use  it  as  ouput  of  the  whole  machine,  as  shown  in  Fig.  2b. 
Note  that  the  linear  transformations  in  front  and  backstrokes 
are  the  transpose  of  each  other,  just  as  in  the  BAM  [5],  In 
hardware,  the  BLT  is  conveniently  implemented  by  a  bidirec¬ 
tional  device  capable  of  linear  transformations  of  vectors, 
such  as  the  Goodman  optical  matrix-vector  multiplier  [17],  or 
an  equivalent  electronic  device. 


The  connection  matrix  of  the  BLT  is  chosen  as 


(3) 


where  q  ,  o(-l  to  N,  are  the  stored  vectors,  and  h  are 

Hadamard  vectors.  In  (3) ,  i  and  b  are  component  indices.  We 
have  chosen  the  same  dimension  N  for  the  BLT  front  and  rear 
vector  spaces,  and  have  taken  the  number  of  stored  states 
equal  to  N  as  well.  It  will  be  clear  from  the  theory  how  to 
modify  these  choices  if  desired.  The  structure  (3)  of  the  BLT 
connection  matrix  B  is  Heobian,  i.e. ,  it  can  be  built  up 
adaptively  by  Kebb  learning.  All  adaption  occurs  in  the  3LT ; 
the  DLS  is  a  fixed  device. 


The  Hadamard  vectors  h  are  rows  of  a  Hadamard  matrix, 

i ■ e • ,  an  orthogonal  matrix  (up  to  a  scalar  factor)  with 
entries  +  or  - .  Material  on  Hadamard  vectors  used  in  this 
report  is  shown  in  the  Appendix.  The  Hadamard  vector  "Tf” 

serves  as  a  label  for  the  stored  state  q*  .  In  its  front 

oC 

stroke  the  BLT  t akes  the  N  stored  states  q  into  certain 

o< 

linear  combinations  of  the  orthogonal  vectors  h 

*ne  orthogonality  of  the  h^  allows  for  a  clean  backstroke 

tnrough  the  BLT,  in  which  a  single  label  h^  produces  a  single 
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stored 


state 


The  action  of  the  SRM  is  as  follows. 


Let  the 


input  "x  be  such  that  it  is  closest  to  stored  vector  q  .  The 
BLT  forward  stroke  gives,  by  (1)  and  (3), 


(4) 


where  c  =  x.q  .  c^  is  the  largest  among  the  coefficients  c  , 
^  o<  I*  & 

0^=1  to  N.  The  vector  u  is  presented  to  the  input  of  the  DLS , 
which  selects  from  the  linear  combination  (4)  the  Hadamard 
vector  ^  which  appears  with  the  largest  coefficient.  The  DLS 

returns  h^  to  the  BLT,  which  performs  the  backstroke  (2)  with 


the  result 


i  L  o  at  C  f-Ot  t  t 

w  =  h  B/  =h.  hy  q  =  N  cJ  q  =Nq„ 

ft,  t,  b  \i 


(5) 


where  use  has  been  made  of  (3}  and  (A2).  Thresholding  with 
the  signum  function  s  then  gives 

.  .  • 

x  '  l'=  s  (w6  )  =  q  6  , 

which  is,  indeed,  the  stored  vector  that  is  closest  to  the 
DLS  input  x*  . 


(6) 


Hence,  if  the  DLS  works  as  required,  the  SRM  performs 
perfect  associative  recall  of  anyone  of  N  stored  vectors. 
There  are  then  no  spurious  stable  states,  and  the  memory  can 
be  loaded  up  fully  to  the  dimension  N.  The  question  is,  of 
course,  how  to  construct  a  DLS  with  the  required  performance. 


DOMINANT  LABEL  SELECTOR  (DLS) 

The  DLS  must  select,  from  an  input 


u  =  c  h  / 

b  *  y 


(7) 


the  dominant  Hadamard  vector  hi  ,  i.e.,  the  index  is  such 

(b  l 

that  c  is  maximum  among  the  coefficients  c  in  (7).  Of 
w  *>( 

course,  h^  is  then  also  the  Hadamard  vector  with  the  largest 
scalar  product  u •  h^  .  Therefore,  the  DLS  itself  may  be  con¬ 
sidered  an  associative  memory  with  stored  states  h  .  In  those 

v(, 

terms,  the  DLS  is  to  produce,  from  the  input  u  of  (7),  the 
closest  stored  state,  TT^. 

One  may  think  that,  since  the  stored  vectors  are 
orthogonal,  a  Hopfield  memory  will  do  fine  in  this  case.  That 
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is  not  so.  The  Hopfield  matrix  [1]  would  be 

^  o(  !o 

S  =  h  h  -nJ  ,  (8) 

4,  a.  ci  u. 

but  with  (Al)  this  is  zero.  Leaving  off  the  term  N cf  does 

s*  J*  . 

not  help,  since  h  h  is  N  times  the  identity  operator,  for 

A  ct 


which  any  vector  is  an  eigenvector.  The  same  results  are  found 
for  any  other  complete  orthonormal  set  of  stored  states,  from 
the  Spectral  Theorem  [16]. 


properties  may 


We  have  found 
be  obtained  by 


that  a  DLS  with 
using  a  quadrat 

b  c 

y  y  +  cr 


the  required 
c  activation, 

(9) 


where  the  connection  tensor  is  fully  symmetric.  The 

last  cerm  in  (9)  may  be  seen  either  as  external  coupling  to  a 
vector  r  with  coupling  strength  c,  or  as  defining  the 

thresholds . 


Neurons  with  activation  (9)  are  a  special  case  of  the 
higher  order  neurons  considered  in  the  literature  [9-12],  In 
the  present  discrete  model,  the  upgrading  of  the  output 
signal  of  neuron  b  from  y^  to  y^  is  done  asynchronously 

according  to  the  assignment 


) 

) 

)  (10) 
) 

) 


E=-  ( 1/3  )  S  .  y  *y  ^yC-cr  yft,  (11) 

&  be  **- 


y'^=  +  if  v,  >0, 


Y*>-  % 

The  "energy"  may  be  taken  as 


=  Ya  if  vz=0, 


4 


if  v.  < 0  . 

b 


which  is  a  simple  extension  of  the  Hopfield  energy  [1]. 
Hopfield 1 s  argument  showing  that  the  energy  cannot  increase 
in  the  discrete  model  is  easily  modified  [10]  for  the  case  of 
quadratic  activation,  as  follows. 


For  a  single  Hamming  step  in  the  direction  of  the 

Pth  coordinate  the  energy  (11)  changes  by 
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The  Hamming  step  C*Y  in  the  direction  of  coordinate  p  may  be 
bitten  f  r«_ 

V  =2ef  r  '  (13) 

where  the  sign  factor  e  =+-l  must  be  chosen  such  that  the 

r 

_li 

#y  keeps  y  in  the  hypercube.  This  requires  that 

V~y  <14) 

rft.  r  r 

In  (13)  ck,  is  the  Kronecker  delta,  so  that  e)  =1  if  p=a ,  and 

r  f 

otherwise  zero.  With  (13)  and  (14),  the  energy  increment  (12) 
due  to  a  Hamming  step  in  the  p  direction  may  be  written 

iE=  -  W  fre-*s«{f  y*  ( 8/3>£sm -  °  (10) 

Substituting  the  activation  (9)  gives 
<5  E=-v* <Jy  -4S 


f 


Y*"+  (8/3)S 


IP 


(ie; 


According  to  the  upgrade  scheme  (10),  v 


s  either 


positive  or  zero,  so  that  the  first  term  in  (16)  cannot  be 
positive  in  the  asynchronous  upgrading.  Hence,  if  the  connec¬ 
tion  tensor  is  restricted  such  that  its  components  with 


at  least  two  indices  the  same  are  zero,  i.e., 

S«^b=0  '  ^a'  '  U?) 

then  the  last  two  terms  in  (16)  vanish,  and  we  have  the 
result 


Theorem  1 :  The  energy  (11),  with  S  subject  to  (17),  cannot 
increase  in  asynchronous  discrete  upgrading. 

This  result  is  known  [10],  and  it  shows  that  the  memory  with 
quadratic  activation  with  the  connection  tensor  S  discussed 
is  stable  in  the  asynchronous  discrete  model. 


In  addition  to  stability  we  require  the  absense  of 
spurious  stable  states,  something  that  is  much  harder  to 
prove.  We  have  such  a  proof  for  the  case  that  the  connection 
tensor  is  given  by 


S  /  =  /  h  h  ,  h  -N  cT,  £ 

dbc  o<a  G-b  <2/ 


cT 

C  A.  t 


-NoT 

c« 


+  2NcT 

<  t 


£,<£,■  <1B 


where  ly  are  Hadamard  vectors.  The  first  term  on  the  r.h.s. 


of  (18)  has  a  strict  Hebbian  form;  the  last  four  terms  have 
been  added  in  order  to  satisfy  the  condition  (17),  while 
retaining  full  symmetry.  With  (18),  the  activation  (9) 
becomes 
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where 


h 

CL 


-2Ny  y  -(N  -2N)cT  +cr 

A  t  CL.  I  CL 


y 

cl 


(  19) 
(20) 


From  (20) 


where  the 
used .  (21) 
important 


we  have 


rZ  Z 

'4rH  ■ 


orthonormality  (Al)  of 
and  the  fact  that  T?, 


consequence  for  a=l 


i 

in 


Hadamard 
has  all  + 
(19)  : 


(21) 

vectors  has  been 
components  has  an 


Choosing 
v(  of  ( 22 ) 


v,  =  cr  . 

cr,  >0 

is  positive,  and  hence,  by  (10), 


(22) 

(23) 


y,  =+  •  (24) 

(24)  means  that  only  half  of  the  hypercube  I.  ={-1,1}*'  is 
dynamically  accessible  to  the  DLS .  In  hardwa're ,  (24)  is 

implemented  by  omitting  neuron  a=l ,  and  by  connecting  the  y 
signal  line  to  the  logicaj  "on"  voltage.  * 


The  energy  increment  (16)  due  to  a  single  Hamming 
step  in  the  p  direction  may  be  written 

Je=-v V  y  ,  (25) 

f  p  *• 

where  (17)  has  been  used.  Because  of  (24),  steps  in  the  1 
direction  are  not  considered: 

P*1  '  (26) 

The  form  (25)  for  the  energy  increment  is  reminiscent  of  the 
energy  increment  in  thermodynamics,  y  and  v  being  respec- 

tively  intensive  and  extensive  variables.  This  typification 
of  variables  is  consistent  with  the  observation  that  as  the 

neural  net  is  scaled  up  in  size,  the  signals  y  keep  their 

A- 

magnitude  range,  while  the  activations  generally  scale 
upward,  due  to  the  increase  in  number  of  signals  to  be 
summed . 


* or  future  use  we  write  the  Hadamard  transform  *Ty 


of  the  step  cT y 

I8  jy  =h  Vy 

j*  oL  c*  fa  cl 

with  (13),  (27)  may  be  written 


(27) 


J 


Y 

OL 


=  2e  h 


/>  V 


(28) 
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STABLE  POINTS  OF  THE  DLS 


We  proceed  to  find  the  stable  points.  Calling 

q  ^2"  y*  h  , 

^  oc  fft  ^ 

and  using  (19),  (13),  and  (24),  Eq.(25)  may  be  written 


a#l,  o E=-2e  (Q 

cl  4 

An  alternate  form  for  (30)  is 
af  1 ,  cT E=-2e  (Q 


<cT E=- 2e  (Q  +cr  -2Ny  )  . 

CL  <£  (5C. 


(29) 


(30) 


c/~E=-2e  (Q  +cr  ) -4N  , 


which  is  derived  by  using  (14). 

(21)  and  h^=+,  \/q(  ,  have  a  consequence  for  a=l  in  (29): 


Q  =N 

I 


(32) 


In  preparation  for  a  determination  of  the  energy  minima  we 
calculate  bounds  on  Q  .  Define  the  index  set 


A=  (oc  |  y  $0  } 
oC 

t 

(33) 

disjoint  pieces 

A  =  €  A  & 

+ 

ii 

(34) 

and 

A  =(o^lo(£rA  & 

i 

ii 

* 

(35) 

is  understood 

that  the  sets 

A*  and  A*" 

depend  on  the 

index  a.  (29)  may  then  be  written  as 

V  z  -z  'C 

JTa-  * 

One  also  has  ^  y  ^  =  N  ^  , 


as  can  be  shown  from  (20),  the  bipolarity  of  y)  and  the 

1  i ty  of  the  Hadamard  vectors.  (37)  may  be  written 


(36) 

(37) 


(38) 

2/  y,  =  N  +  q„  , 

(39) 

a.  2 

2/  Y.  =  N  -  Q  , 

»  —  p(  CJ 

(40) 

^  €.A~ 


and  since  the  left  hand  sides  are  nonnegative,  this  gives 


X.  X 

We  need  to  clarify  the  attainment  of  the  bounds  -N  and  N  in 
(41).  Anyone  of  these  bounds  is  attained  by  Q  ,  for  some 

— k 

index  a,  at  the  Hadamard  points,  with  the  exception  of  h^  . 
This  may  be  seen  as  follows.  For  "y=h*  one  has  y^=0, 
and  y  =N.  (29)  then  gives 

(b 


2. 

Q  h 

A  (2>* 


For  ^>  =  1  we  have  h(  ^=+ ,  ^  a ,  and  it  follows  that 


=N 


Y=h 


Q  «N*, 


(42) 

\f  a  : 

(43) 


For  ^ 

that 

1  jl ,  has  + 

as  well  as 

— 

(b±  /  ■= 

_7  a 

and 

(**'  — 

3  a 

(42) 

further  shows 

that 

Q  =N 

I 


V(2> 


(43) 


Hence,  when  discussing  the  case  Q  =-N  ,  a=l  needs  to  be 

excluded.  „  <K 

For  Q^=-N  the  left  hand  side  of  (39)  is  zero,  and 

hence  the  set  A4  is  empty.  This  implies  that  h  =-  for  all 
in  A.  Therefore  we  have 


af  1  , 


and 


VN 

Q  =N  * 

A 


h  =+ 

A 


>  £  A 


(45) 

(46) 


as  follows  from  a  similar  argument.  The  converse  of  (45)  is 
also  true,  since  from  h  =- ,  V  oi  G  A ,  it  follows  that  a4l 
and  __  „  „ 

Q  ="^-L  =  _N  ' 

CL  ^  D< 

where  (37)  has  been  used.  Hence  we  have 

.2 


Lemma  1 :  a#l , 


Q  =-N 

Cl 


h  =-  ,  Vof  &  A. 

P<  A. 


Next,  we  calculate  the  Hadamard  components  y  given  by  (20). 
Define  the  index  set 
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3=<b|y  =-}  . 

l>  V"  ^  V  b 

Then  we  may  write  (20)  as  y  =  h  y  =-/  h,  +/  h  . 

«  *  *  be S  m 

The  Hadamard  vectors  chosen  have  the  property  (A8): 

Zh4=  NcT,  . 


It  follows  that 


Zh>  ncT-J 


Ix/Pb 


V 


so  that  (48)  yields 


h / 

<■( 


=  N  <T  -2  7 

r  ex'/  /— 

4eB 

Suppose  Q  =-N  ,  \/a€.B.  Then,  Lemma  1  and  (51)  give 

CL 

e><  <r  A  : 


y  =NcT  +2  (  N-W )  / 2  =  NcT  +  N-W 

c<  o <(  ex  l 


where 


(47) 

(48) 

(49) 

(50) 

(51) 


(52) 

(53) 


is  the  weight  of  the  bipolar  vector  y,  and  (N-W)/2  is  the 
cardinality  of  the  set  3. 

Hence  we  have 


Lemma  2  :  Q  =-N  ,  V^a^B 


y  =NcT  +N-W,  \/o<  £  h 

«•<  <*  I 


There  appear  to  be  two  cases: 

Case  l:o<=l  €  A,  and  Case  2:<x'  =  l  ^  A 


Case  1:  o<-l  6  A.  Calculating  y  of  Lemma  2,  we  have 
y( =2N-W=w ,  from  (20),  (53),  and  the  fact  that  h^  has  all  + 

components.  It  follows  that  W=N,  so  that  "y=h(  ,  the  all  posi¬ 
tive  vector.  However,  for"y=h°,  one  has  Q  by  (43),  in 

a  1  **- 

contradiction  with  Q  =— N  of  Lemma  2.  Hence,  Case  1  is  not 

CL 

possible  within  the  premises  of  Lemma  2. 

Case  2:  o(  =1  ^  A.  From  (20)  for  o<  =  l  and  (53)  it  follows  that 

W=0.  Hence,  forc^+l,  the  y  of  Lemma  2  is  just  N.  With  (37) 
it  follows  that  ^  .  « 

N  V  Xrf  =rN^  '  (54) 

where  r  is  the  cardinality  of  the  set  A.  Since  (54)  implies 
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r=l ,  the  set  A  contains  only  a  single  element,  say /i  .  It 
follows  that  "y=hp>  ,  and  with  Lemma  2  we  may  conclude 

=  -N2  ,  Va<£ 


B  y  is  Hadamar d^h- 

_j  _ _ & 

The  converse  is  also  true,  since  for  y-h^, »  we  have  y^=0, 

,  and  y^=N,  so  that 

Z 

(55) 


2 

Q  =N  h 

CL  (b  4 


For  index  a  such  that  y  =h„  =-  it  follows  that  Q  =-N  . 

k  (be  CL 


Hence  we  have 


Theorem  2:  Q  =-N^,  V"a  ^ 


y  =- 

CL 


y  is  Hadamard^h 


With  Lemma  1,  this  theorem  also  can  be  put  in  the  form 


Theorem  3:  h.  =-,  VcxLfiA  and  V&  3  v  =- 

o<  C  CL 

— •■A 

— — 1 *•>  y  is  Hadamard^ 


The  condition  y  =-  m  Theorems  2  and  3 

CL 

SL 

(14),  Using  Q  =-N  and  e  =+  in  (31) 
CL  <*■ 

Je=2N3  -2cr  -4N 

A. 


implies  that 
gives 


by 


(56) 


We  want  to  choose  the  threshold  arrangement,  com¬ 
prised  by  c  and  Tr*  ,  such  that  a^E  of  (56)  is  positive  for  all 

a  involved,  i.e.,  a  such  that  e  =+ .  This  is  the  case  if 

cr  <N*-2N  ,  (57) 

and  N>  2  ;  (58) 

the  dimension  N  is  henceforth  restricted  in  this  manner. 


We  wish  the  left  hand^ side  of  (57)  to  be  independent  of  a; 
this  is  the  case  if  r  is  chosen  as  the  all  positive  bipolar 

vector,  r*  =  hf  .  (59) 

With  this  choice,  (57)  becomes 
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c<N  — 2N 


(60) 


For  the 


Hamming  steps  considerec  above  we  have  ea_~+ • 


It 


remains  to  investigate  steps  with  e  =- ,  a^l , 

a 

Hadamard  point.  For  y=h.~  ,  Q  =N  h  .  By  (14), 

{>  £3L  (b<l 

with  e  =-  we  need  y  =h  f,  and  hence  Q  -N  . 

*  a_  j&  t*. 

(30)  gives  r  i 

a}l,  cJE=2(N  +C-2N) 

A. 


made  from  a 
for  a  step 
Substitution  in 

(61) 


We  want  the  energy  increment  (61)  to  be 
all  steps  considered  in  this  case,  i.e. 
This  requires  that  ^ 

c  >  -(N  -2N ) 


positive  as  well,  for 

a  such  that  e  =- . 

* 

(62) 


(30)  and  (62)  show  that  stability  of  the  Hadamard  points 
requires  the  common  threshold  c  to  satisfy  the  condition 

2.  a, 

-(N  -2N ) <  c  <  N  -  2  N  .  (63) 


The  upper  bound  is  positive  if  (58)  is  satisfied.  With  (23), 
(59),  and  the  fact  that  has  all  positive  components,  we 
can  sharpen  (63)  to 

7 

0<c<N  -2N  .  (64) 


Hence  we  have 


Theorem  4 :  For  a  quadratic  Hadamard  memory  with  asynchronous 
discrete  dynamics  given  by  (9)  and  (10),  with  connection 
tensor  (18),  and  a  threshold  c  subject  to  (64),  the 
Hadamard  points  are  stable  equilibria. 


It  remains  to  be  shown  that  a  threshold  c  subject  to  (64) 
exists  such  that  there  are  no  other  stable  points  besides 
Hadamard  points.  This  is  done  by  showing  that  the  possible 
values  of  have  a  gap  of  magnitude  4N  at  the  lower  bound 

Z 

-N  .  With  (41)  this  implies  that  there  is  a  gap  of  magnitude 
8N  at  the  upper  bound  of  the  energy  increments  cTe.  This  means 

OL 

that  the  threshold  c  can  be  chosen  such  that  the  energy 
increment  gap  straddles  the  origin.  We  then  have  the  desi¬ 
rable  situation  that  for  Hadamard  points,  and  only  for  those, 

cTe>0  Vafl;  for  any  other  points  "y*  there  always  is  an  index 
CL 

a  such  that  ^E<0  .  This,  of  course,  shows  that  the  Hadamard 
* 

points  are  the  only  stable  points  of  the  DLS .  We  proceed  with 
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the  details  of  the  outlined  steps. 

By  Theorem  2 ,  for  y  Hadamardfch(  and  for  Hamming  steps  in 

the  direction  a  such  that  e  =+  (and  therefore  y  =-,  by  (14)) 

A-  CK 

A 

we  have  Q  =-N  .  Writ:  lg 
a. 

a  7 

c=N  - 

(56)  gives 


-K,  0<K< 2N  -4N 

t 

(65) 

Je=2K, 

(66) 

ct 

Vice  versa, 

Q  =  -N  y  is 

Hadamard4h  , 

(67) 

c>  E=2K,  Va  3-  e  =+ 

where  use  has  been  made  of  (14),  (31),  (59),  and  Theorem  2 

Hence  we  have 


Theorem  5 :  y  is  Hadamardfh, 


t>  cf  E=2K ,  Vz 
CL 


'a  3  y  =-  . 

a. 


(68) 


Next,  we  calculate  the  increment  /q  of  Q  ,  a=fl,  due 

ct 

to  a  single  Hamming  step  cTy  in  the  direction  p£l .  From  (29) 
one  has  -  _ 

4  y  <£?  +  • 

A3  O-  ^  *=<  f>en  fiy'  e<c* 


(69) 


With  (28),  the  first  term  in  (69)  may  be  written 

2  /  _  y  2e  h  h  =  4e  y  h  h 


(70) 


The  group  property  of  the  Hadamard  vectors,  discussed  in  the 
Appendix,  makes  the  (component-wise)  product  of  two 
Hadamard  vectors  a  Hadamard  vector,  i.e., 


V  *u =  W  •  • 


(71) 


w 

where  c=f(p,a);  see  (A16).  For  the  present  purpose  we  do  not 
need  to  know  what  this  function  is.  Since  our  proof  of  the 
group  property  given  in  the  Appendix  requires  that  the  dimen¬ 
sion  is  a  power  of  2 ,  N  will  henceforth  be  restricted  in  this 
manner.  Furthermore,  in  order  that  (58)  be  satisfied,  the 
power  is  restricted  to  be  at  least  2.  Using  (71)  in  (70) 
gives 

•  <7?> 

by  applying  the  inverse  of  the  Hadamard  transform  (20). 

With  (28)  and  (A7),  the  second  term  in  (69)  may  be  written 
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( 2e,  h  )  h  =4/  h  =0 


(73) 


21  fc4  ^  =Z< 

since  a+1  .  (72)  and  (73)  give  for  the  increment  <Tq  of  (69) 

r  ^ 

a+1,  ptl,  e>Q  =4e  Ny  .  (74) 

pa.  (>  c 

which  shows  that  the  only  possible  values  for  <Jq  are  +-4N. 

fi*- 

With  Theorem  2,  (41) ,  and  the  fact  that  any  of  the  two  bounds 

a.  a. 

N  and  -N  in  (41)  are  attained  by  Q  for  some  af  1  ,  it  follows 

A 


that  for  a^l  the  range  of  is  included  in  the  set 

o.  % 

( -N  , -N  +4N,-N  +8N ,  ...  , N  }  . 


(75) 


With  (31)  and  (66)  this  implies  that  the  range  of  energy 

increments  <f  E  is  contained  in 
a 

a. 

{ 2K ,  2K-8N,  2K-16N,  . . . 2K-8N  }  .  (76) 

(76)  shows  that  the  energy  increment  gap  is  8N.  The  gap  may 
be  placed  symmetrically  over  the  origin  by  choosing  K  such 
that 

2K-8N=-2K  ;  (77) 

this  is  the  case  if  K=2N  .  (78) 

With  (78)  and  (65)  we  have  for  the  threshold 

2 

c=N  -4N  .  (79) 

The  symmetric  placing  of  the  energy  increment  gap  maximizes 
fault  tolerance  of  the  associative  memory. 

With  (78),  the  energy  increment  range  is  contained  in  the  set 

{  4N ,  -4N  ,  -12N,  .  .  .  ,  4N-8n':S  .  (80) 

Theorem  5  together  with. (78)  show  that  only  at  Kadamard 
points  the  increments  c)e  are  positive  for  all  steps  with 

e  =+.  For  other  points  y,  there  always  is  a  Hamming  step 

which  gives  a  negative  energy  increment.  With  Theorem  1  it 
follows  that  such  points  are  not  stable.  Hence,  we  have 
proved 

Theorem  6:  For  the  asynchronous  discrete  quadratic  Hadamard 
memory  with  connection  tensor  (18)  and  threshold  (79)  the 
stable  points  are  just  the  Hadamard  points. 

Evidently,  there  are  no  spurious  stable  states  in 
the  quadratic  Hadamard  memory  considered.  In  contrast,  for 
an  N-dimens ional  Hopfield  memory  with  N  stored  states,  all 
taken  as  Hadamard,  every  state  is  a  spurious  stable  state, 
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excepting  of  course  the  Hadamard  states  themselves. 

It  follows  that  an  N-dimensional  SRM  with  the  DLS  taken 
as  a  quadratic  Hadamard  memory  has  perfect  associative  recall 
of  up  to  N  bipolar  stored  vectors.  The  latter  can  be  chosen 
arbitrarily.  There  are  no  spurious  states. 


ALTERNATE  SYSTEM 

There  is  a  variation  on  the  system  considered  above, 
which  gives  the  same  main  results.  In  the  ongoing  research 
we  stumbled  first  upon  the  alternate  system,  and  much 
computer  work  was  done  in  its  exploration. 

In  the  alternate  system  the  connection  tensor  is 
taken  to  have  the  strictly  Hebbian  form 


=  /  h  h  ,h  , 

*  tr  *<* 


(81) 


without  any  of  the  subtractions  of  (18).  The  activation  is 
then  simply 

v  =Q  +c  ,  (82) 

where  (59)  has  been  used.  With  (43)  we  have 

Cl 

v  =N  +c  ;  (83) 

if  c  is  restricted  by  (64),  then  v  is  positive,  and  thus 

Y  =+  ,  (24) 


as  for  the  subtracted  tensor. 

The  energy  increment  (16)  involves  the  terms  with  S 


tr 


and  S. ,  ,  which  must  be  calculated,  since  the  connection 

tensor  is  "unsubtracted" .  For  these  terms  we  find  for  pfl 


y-Z 

o( 


r  *■ 

d  v  =-' 


h  y  =  -4N  d  y  =-4Ny  =-4N 

At  I 


—  4S  .  y *  =-4  7~  h,  h  ,h 
by  ( A7 )  and  (24),  and 

<8/3>yvP  ■'W'vfZivf)J=(8'3)v ^^V(8/3)VNi'‘0, 

since  p=fl  .  Hence,  for  the  unsubtracted  connection  tensor  (81) 
and  for  p^l ,  (16)  becomes 

p4=l  ,  c>E=-v',c>y  -4N  .  (84) 

A  **  ?  A 

Since  v  4 y  > 0  by  (10),  and  the  extra  term  is  negative,  we 
have  »  A 
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Theorem  1 1 :  For  the  quadratic  Hadamard  memory  with  unsub¬ 
tracted  connection  tensor  (81),  the  energy  ( 1 1 )' cannot 
increase  in  asynchronous  discrete  upgrading. 


The  increment  of  the  energy  (11)  due  to  a  single  Hamming 
step  in  the  direction  a+1  may  also  be  written 

a*l,  J E=-2e  (Q  +c)-4N  ,  (85) 

(l  «.  A 

where  (82)  has  been  used.  Since  (85)  is  identical  to  (31) 
with  (59),  the  arguments  leading  to  Theorem  6  go  through 
for  the  alternate  system,  and  we  have 


Theorem  6 1 :  For  the  asynchronous  discrete  model  of  the 
quadratic  Hadamard  memory  with  connection  tensor  (81)  and 
threshold  (79),  the  stable  points  are  just  the  Hadamard 
points . 


COMPUTER  SEARCH  FCR  STABLE  POINTS 

As  a  check  on  Theorems  6  and  6'  v.e  have  performed  a  search 
for  stable  points  y  for  a  discrete  quadratic  Hadamard  memory 
of  dimension  ^6 .  The  program  involves  a  search  for  equili¬ 
brium  jDoints  y  of  the  hypercube  I(^  ={-l ,  1 }  .  For  each  such 

point  "y  the  activations  (9)  of  the  15  neurons  a+l  are  calcu¬ 
lated,  and  (10)  is  used  to  see  if  upgrading  causes  any  change 
in  the  signals  y?-  If  there  is  no  change,  then  ^  is  an  equili¬ 
brium  point  of  the  quadratic  Hadamard  memory  considered.  If  y 
is  found  to  be  an  equilibrium  point,  a  subroutine  is  called 
which  computes  the  energy  (11)  at  ~y  and  at  all  the  neighbo¬ 
ring  points  one  Hamming  step  away  from  y\  If  E(y*)  is  smaller 
than  the  energy  at  each  of  these  neighboring  points,  the 
equilibrium  point  y  is  stable,  in  the  discrete  sense,  and 
the  case  is  reported.  The  search  extends  over  all  64K  points 
of  the  hypercube  I  y  . 

Programs  have  been  run  using  the  subtracted  connec¬ 
tion  tensor  (18),  as  well  as  the  unsubtracted  tensor  (81). 

For  both  cases,  the  computer  runs  show  as  only  stable  points 
the  Hadamard  points,  in  agreement  with  Theorems  6  and  6'. 


CONNECTIONS  IN  THE  QUADRATIC  HADAMARD  MEMORY 

There  is  a  practical  concern  about  the  number  of 
physical  connections  required  in  the  quadratic  Hadamard 
memory.  Physical  connections  are  required  between  neurons  a, 
b,  and  c,  if  40 .  Fortunately,  for  given  by  (18) 
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or  (81),  most  components  are  zero.  This  is  due  to  the  group 
property  of  the  Hadamard  vectors. 

For  the  connection  tensor  given  by  (81): 

S/=/hh.h  (81) 

the  product  of  two  Hadamard  vectors  may  be  replaced  by  a 
single  Hadamard  vector,  by  virtue  of  the  group  property 
discussed  in  the  Appendix: 

-  <86> 

where  d=f(b,c)  is  the  structure  function  of  the  group. 
Substitution  in  (81)  and  using  the  group  property  once  more, 

'  <87> 

e=f ( a , d ) ,  gives 

S  /  =Th  =N/  ,  (88) 

where  (A7)  has  been  used  in  the  last  step.  (88)  shows  that 
Sa(?c  iS  nonzero  onIY  for  e=l.  By  (87)  this  requires  that 

a=d,  (89) 

as  follows  from  the  material  on  the  Hadamard  group  shown  in 
the  Appendix.  It  follows  that  is  nonvanishing  only  if 

a=f ( b , c )  .  (90) 


Hence  we  have 

Theorem  7 :  For  connection  tensor  (81)  the  quadratic 

2 

Hadamard  memory  requires  N  physical  connections;  each  of 
these  has  the  weight  N. 


For  the  subtracted  connection  tensor  (18)  the  number  of 
nonvanishing  connections  is  further  reduced  by  N,  the  number 
of  pairs  (b,c)  for  which  b=c .  Hence  we  have 


Theorem  8 :  The  quadratic  Hadamard  memory  with  subtracted 

connection  tensor  (18)  requires  N  -N  physical  connections;  all 
of  these  have  weight  N. 
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The  quadratic  Hadamard  memory,  serving  as  a  DLS  in  our 
SRM,  achieves  the  desirable  absence  of  spurious  states  by 
using  quadratic  activation.  It  is  fortunate  that  this  can  be 
done  without  the  proliferation  of  physical  connections  which 
usually  accompanies  higher-order  neurons  [9].  The  fact  that 
all  the  nonzero  weights  have  the  same  value  N  very  much 
simplifies  the  DLS  hardware:  the  interconnects  may  be  taken 
just  as  wires,  since  the  factor  N  is  not  essential. 

The  main  complication  facing  us  is  the  implementation 
of  the  signal  product  operation  required  for  computing  the 
quadratic  activation.  Such  signal  multiplication  can  be  done 
with  a  field  effect  transistor  which  is  made  to  operate  in 
the  unsaturated  region.  The  details  of  this  operation  need 
to  be  studied.  It  is  expected  that  considerable  deviations 
from  a  strict  product  can  be  permitted,  because  of  the  fault 
tolerance  of  the  quadratic  Hadamard  memory. 


COUPLING  BETWEEN  BLT  AND  DLS 

The  forward-stroke  output  u  of  the  BLT  must  be 
coupled  to  the  DLS  in  such  a  manner  that  the  DLS  can  extract 
the  dominant  Hadamard  vector  in  tf.  We  have  been  deliberately 
vague  about  this  coupling,  for  the  following  reason. 

The  BLT  forward-stroke  output'll  is  generally  not 
bipolar.  However,  the  BLT  model  used  in  this  report  is 
discrete,  so  that  the  BLT  signal  stately  is  bipolar.  Hence, 
u*  cannot  be  used  as  initial  value  for  "y.  Of  course,  one  could 
threshold  "u  and  apply  the  result  as  initial  y*.  That  would  be 
a  pity  though,  because  in  the  thresholding  the  fidelity  of 
the  Hadamard  expansion  comprising  "u  would  be  lost,  "u  must  be 
coupled  to  the  DLS  in  some  other  way.  We  have  found  how  to 
do  this,  but  the  description  requires  the  continuum  model 
which  allows  y  to  have  as  components  real  numbers  in  the 
range  between  -1  and  1,  and  accounts  for  input  capacitance 
and  resistance  of  amplifiers.  In  this  model,  the  DLS 
dynamics  is  described,  in  normalized  form,  by 

i=-V^+JUc^YC  '  (91) 

Y«.=S(V*)  '  (92) 

where  the  dot  denotes  differentiation  wi  respect  to  time, 
and  s  is  a  sigmoid  function.  The  forward  coupling  between 
BLT  and  DLS  can  be  arranged  such  that  is  applied  as 

initial  value  for  v,  being  a  coupling  constant.  Choosing /*< 
is  a  delicate  matter  which  will  be  discussed  in  a  future 
report  entitled  "Quadratic  Hadamard  Memories  II”.  This 
report  will  deal  with  the  continuum  model  of  the  SRM,  the 
DLS,  and  of  the  entire  system,  SRM. 
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APPEND1X 
Hadamard  vectors 


In  the  report  we  use  bipolar  vectors  which  form 
an  orthonormal  set.  A  matrix,  the  rows  of  which  are  such 
vectors,  is  called  a  Hadamard  matrix  [13].  Such  a  matrix  is 
orthogonal,  up  to  a  scalar  factor  N,  the  dimension.  Hadamard 
matrices  have  an  important  application  in  optimal  multi¬ 
plexing  [13];  they  are  used  in  spectrometers  and  imagers  in 
order  to  improve  the  s ignal- to-noise  ratio.  Optics  employing 
this  method  is  called  "Hadamard  transform  optics"  [13]. 
Hadamard  matrices  are  also  used  in  error-correcting  codes 
[13].  It  is  no"  surprising  that  they  can  be  used  to  advantage 
in  neural  nets  as  well. 


The  rows  of  a  Hadamard  matrix  are  here  called 
"Hadamard  vectors" .  They  are  special  elements  of  the  hyper¬ 
cube  that  form  an  orthogonal  set .  .The  N-dimens ional  Hadamard 
vectors  used  here  are  denoted  by  h  ,  «  =1  to  N.  Labeling  the 

vector  components  by  the  index  b=i  to  N,  the  Hadamard  matrix 

with  rows  h  has  the  elements  h  .  The  orthonormality  of  the 

Hadamard  vectors  is  expressed  by 

xJ  b  r~  h 

n  h,  =N  a  ,  ( A1 ) 

where  is  the  Kronecker.  From  (Al)  and  the  linear  inde¬ 


pendence  of  vhe  Hadamard  vectors  a  second  set  of  orthonorma¬ 
lity  conditions  can  be  derived: 


(A2) 


For  use  in  this  report  we  restrict  the  Hadamard  matrix  to  be 
symmetric,  and  to  have  their  first  components  equal  to  +.  For 
dimensions  N  that  are  powers  of  2  there  are  the  so-called 
"Sylvester-type  Hadamard  matrices"  [13],  which  are  defined 
recursively  by  the  scheme 


(  K 

H  =  (  * 

V 
)  / 

(A3 

H  -  Cl). 

2 

2N  <  V 

-v 

( A4 

For  example,  the  Sylvester- type  Hadamard  matrix  of  dimension 
16  is  shown  below. 
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There  is  a  second  type  of  Iiadamard  matrices.  These 
matrices  are  constructed  from  so-called  cyclic  S-matrices 
[13],  and  they  exist  only  for  certain  dimensions  N,  including 
all  powers  of  2.  The  construction  of  the  matrix  starts  with 
choosing  the  first  Hadamard  vector  to  have  all  components  +. 
The  remaining  N-l  Hadamard  vectors  are  found  by  taking  their 
first  component  as  +,  and  by  taking  the  remaining  N-l  compo¬ 
nents  as  left  shifts  of  a  special  N-l  dimensional  bipolar 
vector  ,  the  construction  of  which  is  discussed  by 

Harwit  and  Sloane  [13].  They  also  show  a  list  of  such  vectors 
for  several  small  values  of  N.  Some  examples,  taken  from  [13] 


3  -  +  - 

7  - 4-44 

11  - 1 - 4-44 4- 

15  4-4-4 1-4- 4 1- - 

The  Hadamard  matrix  constructed  from  z  for  N=16  is 


4-4-4-4-4-4-4-4-4-4-4-4-4-4-4--I- 
4444  —  44  —  —  4  —  4  —  —  —  — 
444  —  44  —  —  4  —  4  —  —  —  —  4 

4-4-  —  4-4-  —  —  4-  —  4-  —  —  —  —  4-4- 

+  —  +  +  —  —  4-  —  4  —  —  —  —  444 

4-4-4-  —  —  4-  —  -f  —  —  —  —  4  4  4  - 

4-4-  —  —  4-  —  4-  —  —  —  —  4-4-4-  —  4- 

4  —  —  4  —  4  —  —  —  —  444  —  44 

4-  —  4-  —  4-  —  —  —  —  4-4-4-  —  4  4- 

4-4-  —  4-  —  —  —  —  4-4-4.  —  +  4-  —  — 

4-  —  4-  —  —  —  —  4-4-4-  —  4-4-  —  —  4- 

+  4-  —  —  —  444  —  44  —  —  4  — 

4----444-44--4-4 
4---444-44--4-4- 
4-  —  —  44-4  —  4*  4  —  —  4  —  4  —  — 

4  —  44  4  —  4  4  —  —  4  —  4'  —  —  — 


(A6) 
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One  has  (see  [13]), 

Theorem  A1  :  If  N  is  a  power  of  2,  the  ’hudamard  matrix  con¬ 
structed  from  a  cyclic  3  matrix  can  be  transformed  into  a 
Sy lvester-type  Hadamard  matrix  by  permutation  of  rows  and 
of  columns. 


This  theorem  is 
below.  For  the 
have 


mentioned  here  because  it  will  be  needed 
Hadamard  vectors  h^  used  in  this  report,  we 


(A7) 


by  the  symmetry  of  the  Hadamard  vectors  used  here,  this  may 
also  be  written  as 

ex. 


Zh  =N  cT 


(A8) 


Group  property  of  Hadamard  vectors 


For  dimension  N=2  ,  k  natural,  we  have  the  Sylvester 
construction  of  Hadamard  vectors,  as  discussed  above.  The 
component-wise  product  of  any  two  such  Hadamard  vectors  is 
a  Hadamard  vector:  L  L  L 

hfh  =hi*  ,  Vb  ,  (A9) 

*  C*  o 

where  <Jf=  f  (e(  ,^> )  .  This  may  be  shown  by  induction,  as  follows. 

The  property__holds  for  N=2 ,  since  by  ( A9 )  we  have  then 
h(=++,  and  h^ ~+- ,  and  (A9)  is  true,  as  can  be  seen  by 

inspection . 

Assume  that  the  group  property  holds  for  Sylvester-type 
Hadamard  vectors  of  dimension  N,  a  power  of  2.  Construct  a 
2N-dimensionciI  Hadamard  matrix  by  means  of  the  Sylvester 
construction . 


H 

2N 


1 

1 

1 

H 

1 

1 

1 

H 

1 

1 

! 

1 

1 

N 

1 

1 

1 

N 

1 

1 

1 

1 

1 

H 

1 

1 

1 

-H 

1 

1 

1 

1 

1 

N 

1 

N 

1 

i 

1 

1 

1 

(A10) 
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where  H  is  the  Sylvester-type  Hadamard  matrix  of  dimension  N. 

.  Jo  L 

Consider  an  index  pair  t* ,p>  to  be  used  in  the  product  h^  fj^on 

the  left  side  of  (A9).  We  have  the  following  cases 
( see  Fig .  3 ) : 

1  )<?<•<  N  .  This  means  that  the  Hadamard  vectors  h^  and  h^ 

both  lie  in  the  upper  half  of  the  matrix  (A10).  Since  the 
group  property  holds  for  H  ,  3  Y  <  N  9{A9)  holds  for  b=l 


/V 

to  N.  But  for  the  upper  half  of  H 


the  matrix  H  of  the  left 
A/ 


half  is  repeated  on  the  right,  hj*  =  h^^t  b=l  to  N.  It  follows 

o(  of 

that  ( A9 )  is  true  for  all  b=l  to  2N. 

2)  c*>n,  |3>,>N.  Hadamard  vectors  h^  and  h  then  lie  both  in  the 

lower  half  of  H  .  Since  the  group  property  holds  for  dimen- 

sion  N,  3r>  N  9 (A9)  is  true  for  b=l  to  N.  But,  in  the  lower 
half  of  H  we  have  /  / 

^  h  h***. 

ex'  0< 


Hence,  for  c=N+l  to  2N, 

( 


e  c  c 

h.  h/T=-h„  . 

*  o  r 


Subtract  N  from  t-r  -N.  Then, 

for  b=l  to  N, 


4  4 

=  V '  ' 

and  for  b=N+l  to  2N,  h  .  =-h  /  ,  because  of  the 

0  0 

block  structure  (A10),  It  follows  that  for  all  b=l  to  2N  we 

have  4  4  4 

.  h^  =h^  /  ;  hence,  (A9)  is  true  for  dimension 

2N  in  this  case. 


7*  T 

3)  ^N,  N.  Let  o(  =N+  .  Then  ,  J7  )- >  N 

for  b=l  to  N, 


such  that 


Let^=^-N.  Then, 

and  (All)  gives 
For  c>N,  we  have 


b  <N, 


h  S b  h 
« '  (b  ir 

4  4 

h  7=h 


(All) 


cx 


,  4 

Cr  * 

h-  ^ -hr '  • 
■ 

,  C  C  ,  C 

h  y  h  =h,  /  . 

P  t 


( A1 2  ) 


and  (All)  gives 


(A13) 
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(A12)  and  (A13)  give 

ts.  a 

a=l  to  2.N,  h  ^  h  _  =h  /  .  (A14) 

*  (*  <f 

Since  the  product  in  (A9)  is  commutative,  the  remaining  case, 
ckp>N,/3kN  reduces  to  case  3). 


Hence,  the  group  property  (A9)  holds  for  dimension  2N.  By 
induction  it  follows  that  the  property  holds  for  all 
Sylvester-type  Hadamard  matrices. 

In  the  DLS ,  we  h?  »/e  used  the  symmetric  Hadamard 
matrices  constructed  from  cyclic  S  matrices.  By  Theorem  A1 , 
if  N  is  a  power  of  2,  these  matrices  can  be  transformed  to 
Sylvester-type  matrices  by  index  permutations.  We  just  have 
shown  that  for  the  latter  type  Hadamard  matrices  the  group 
property  holds.  Index  permutations  do  not  affect  the  group 
property.  Hence  we  have 

Theorem  A2 :  For  dimensions  that  are  a  power  of  2,  the 
Hadamard  vectors  constructed  from  cyclic  S  matrices  form  a 
group  under  component-wise  multiplication. 

We  call  the  group  under  discussion  the  "Hadamard 
group".  In  this  group,  the  first  Hadamard  vector,  which  has 
all  components  +,  acts  as  the  identity.  Every  Hadamard  vector 
is  its  own  inverse.  The  function  y-=f  [o(  ,(b)  which  gives  the 
"product  index"  V  for  any  pair  of  "factor  indices"  may 

be  called  the  "structure  function"  of  the  group.  Since  the 
component  multiplication  in  (A9)  is  commutative,  the  Hadamard 
group  is  abelian. 

Since  the  Hadamard  matrices  used  here  are  symmetric, 
the  group  property  may  also  be  expressed  as 

h?h*=h°  ,  ,  (A16) 

CX  QC 

with  c  —  f ( a , b ) . 
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R  i  9  •  I  •  Select  i  ue  Ref  !  e:<  i  ue  f '  e  m  o  r  y  (  SRP ) ,  consist  i  og  of  a 
Bidirectional  Linear  Trans  former  (8LT)  and  a  Dominant 
Labe!  Selector  (DLS).  The  SRH  takes  a  bipolar  input  X  to  a 
bipolar  output  X  .  The  8 L T  processes  data  in  a  f o r ui a r d 
st  rok  e :  X  -»U,  and  a  backstroke  y  -j>  W.  The  DLS  computes 
■  rom  hi  a  quadratic  act  i  oat'  on  V',  which  is  thresholded  to 
the  bipolar  orator  y.  y  *>d  back  to  the  DLS  input.  Rlso 
y  is  processed  by  the  BLT  in  the  backstroke:  y-»W,  and  W 
is  thresholded  to  qiae  the  bipolar  output  x' 


